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SUMMARY 


The  formal  techniques  of  decision  analysis  (DA)  are  a 
procedure  for  improving  decision  making  under  uncertainty. 
Although  the  logical  foundations  of  decision  analysis  have 
been  generally  accepted,  there  is  still  some  skepticism 
among  potential  users  of  the  procedure  about  its  effective- 
ness in  practice.  Several  factors  make  it  difficult  to 
estimate  the  value  of  such  an  analysis  even  after  it  has 
been  completed.  Typically,  because  the  problems  studied  by 
using  DA  involve  particular  decision  situations  which  do  not 
recur  in  the  same  form,  an  analyst  cannot  compare  the  de- 
cision recommended  by  the  DA  with  that  which  would  have  been 
taken  if  no  analysis  were  done  or  if  some  other  type  of 
analysis  were  undertaken.  Since  these  comparisons,  in  fact, 
cannot  be  made,  they  must  be  guessed  at  by  the  analyst. 
Furthermore,  because  of  the  element  of  chance,  the  optimal 
decision  taken  under  the  DA  (the  decision  option  having  the 
greatest  expected  utility)  may  not  turn  out  to  have  the  best 
outcome  when  the  uncertainty  is  resolved. 

Despite  these  difficulties,  however,  it  is  proposed  in 
this  paper  that  a systematic  attempt  be  made  to  evaluate 
completed  analyses.  A methodology  for  this  evaluation  is 
presented.  The  first  step  of  the  evaluation  is  to  determine 
the  general  degree  of  success  of  the  analysis:  Was  the 

analysis  completed  and  was  its  recommendation  followed? 

Next,  the  specific  benefits  and  costs,  both  direct  and 
indirect,  are  enumerated  and  summarized.  Finally,  a critique 
is  made,  of  the  application  of  each  of  the  DA  techniques. 

The  purposes  of  this  evaluation  procedure  are,  first, 
to  encourage  users  of  DA  to  review  briefly  and  evaluate 
their  experience  after  every  analysis  and,  second,  to  pro- 
vide some  general  statistics  about  the  strengths  and  weak- 
nesses of  DA  as  it  is  practiced.  To  gather  these  general 
statistics,  it  is  recommended  that  the  evaluation  procedure 
be  carried  out  for  10  to  20  recent  applications  of  DA  by 
means  of  interviews  with  the  analysts  and  users  as  well  as 
by  consulting  the  written  reports  of  the  analyses.  This 
paper  provides  sample  questionnaires  which  can  be  used  to 
elicit  the  desired  information. 
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1.0  DIFFICULTIES  ENCOUNTERED  IN  THE 
RETROSPECTIVE  EVALUATION  OF  DECISION  ANALYSIS 


The  retrospective  evaluation  of  an  application  of  the 
techniques  of  decision  analysis  (DA)  is  difficult  for  a 
number  of  reasons.  The  formal  procedures  of  DA  cannot  be 
followed  exactly  in  most  practical  settings;  an  analysis, 
even  if  perfectly  applied,  may  have  no  directly  ascertain- 
able benefits  either  because  the  recommended  decision  is  the 
same  as  that  which  would  have  been  taken  without  the  analysis 
or  because  the  uncertainty  is  ultimately  resolved  in  a way 
which  favors  some  other  decision  over  that  recommended  by 
the  analysis.  Furthermore,  the  indirect  advantages  of  a DA 
may  be  difficult  to  express  in  quantitative  terms,  r.s  are 
some  of  the  indirect  costs  of  DA. 

1 . 1 Methodology  of  Decision  Analysis 

Decision  analysis  is  a method  for  selecting  the  best 
available  option  in  decisions  that  involve  some  uncertainty . 1 
The  method  is  a formal  procedure  in  which  the  decision 
problem  is  decomposed  into  several  parts  that  can  be  treated 
separately.  The  results  of  these  separate  treatments  are 
then  aggregated  to  produce  a recommended  decision.  The 
first  step  in  the  procedure  is  the  enumeration  of  all  decision 
options,  relevant  uncertain  events  and  consequences  resulting 
from  combinations  of  decision  options  and  event  outcomes. 

Next,  subjective  probabilities  are  elicited  for  the  different 
outcomes  of  each  uncertain  event,  and  the  decision  maker's 
utility  for  the  various  possible  consequences  is  assessed. 
Finally,  this  information  is  combined  by  computing  an  expected 
utility  value  for  each  option.  The  option  having  the  highest 
expected  utility  is  recommended  by  the  DA. 

The  theoretical  foundations  of  DA  have  been  the  object 
of  considerable  debate  in  recent  years.  The  inquiry  centers 
on  the  use  of  subjective  probabilities  to  express  uncertainty , 2 
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See,  for  example,  H.  Raiffa,  Decision  Analysis  (Reading, 
Massachusetts:  Addison-Wesley^  1968)  ; and  R.  V.  Brown, 

A.  S.  Kahr,  and  C.  R.  Peterson,  Decision  Analysis  for  the 
Manager  (New  York:  Holt,  Rinehart,  and  Winston,  1974). 
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L.  J.  Savage,  The  Foundations  of  Statistics  (New  York: 
Wiley,  1954) . 
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multi-attributed  utility  functions  as  a measure  of  value,3 
and  maximization  of  expected  utility  as  the  criterion  for 
rational  decision  making. 4 At  the  same  time,  a great  deal 
of  experimentation  has  been  carried  out  to  test  different 
techniques  for  actually  eliciting  the  probability  and  utility 
values  required  by  the  DA  procedure?  On  the  bases  of  this 
theoretical  and  experimental  work,  many  people  believe  that 
the  use  of  DA  ought  to  improve  the  decision  making  of  an 
organization  or  an  individual.  It  remains  to  be  shown  in  a 
systematic  way,  however,  that  such  improvements  are,  in 
fact,  realized. 


1 . 2 Practical  Limitations  of  Applied  Decision  Analysis 


Decision  analysis  as  it  is  practiced  does  not  conform 
perfectly  to  the  theoretical  model  of  DA.  Differences 
between  the  actual  and  the  ideal  use  of  the  method  may  limit 
the  value  of  DA.  For  example,  DA  often  simplifies  real 
decision  problems  by  omitting  some  of  the  options  and  uncer- 
tain events  in  order  to  speed  up  the  analysis.  Important 
factors  to  the  decision  problem  may  also  be  inadvertently 
left  out.  Obviously,  the  best  decision  option  cannot  be 
selected  if  it  is  not  included  in  the  analysis.  The  DA  can 
also  suffer  if  the  numerical  inputs,  the  probabilities 
measuring  the  uncertainty,  and  the  utilities  giving  the 
value  of  the  consequences  are  incorrect.  In  most  cases, 
these  figures  will  be  approximations  of  the  quantities  being 
measured  rather  than  exact  numbers. 


The  possible  lack  of  rigor  when  the  DA  methodology  is 
applied  is  one  limitation  to  the  analysis;  another  limitation 
sometimes  observed  is  the  lack  of  acceptance  of  the  recommen- 
dation of  the  DA  simply  because  of  resistance  to  the  new 
method  by  the  people  concerned  with  the  decision.  Such 
resistance  is  often  observed  when  new  methods  of  management 
are  introduced  and  is  especially  prevalent  when  the  techniques 
are  of  a quantitative  or  mathematical  nature.® 


^R.  Keeney  and  H.  Raiffa,  Decisions  with  Multiple  Objectives 
(New  York:  Wiley,  forthcoming!"! 

4 

J.  Von  Newmann  and  0.  Morgens tern.  Theory  of  Games  and  Eco- 
nomic  Behavior  (Princeton,  New  Jersey : Princeton  University 

Press , 1947) . 

5G.  Huber,  "Methods  for  Quantifying  Subjective  Probabilities 
and  Multi-Attribute  Utilities,"  Decision  Sciences  5 (1974): 
430-58. 


^See,  for  example,  C.  J.  Grayson,  Jr.,  "Management  Science 
and  Business  Practice . " Harvard  Business  Review  51  (1973):  44. 


1.3  Statistical  Difficulties  in  Evaluating  a Decision  Analysis 


When  a DA  is  carefully,  even  flawlessly,  performed,  diffi- 
culties can  still  arise  in  the  evaluation  of  the  analysis. 

DA  claims  that,  on  the  average  and  in  the  long  run,  the  results 
of  decision  making  by  this  technique  should  be  superior  to 
those  obtained  by  some  other  method.  A particular  decision 
maker  making  a particular  decision,  on  the  other  hand,  may 
not  be  able  to  show  this  improvement  and  may  not,  in  fact, 
be  better  off  than  with  another  technique. 

First  of  all,  the  DA  may  simply  recommend  the  decision 
that  would  have  been  taken  anyway.  In  this  case,  the  DA  pro- 
vides additional  evidence  suggesting  that  the  selected  decision 
is  the  optimal  one,  and  this  information  may  bring  increased 
confidence  or  peace  of  mind  to  the  decision  maker.  However, 
the  value  of  such  peace  of  mind  could  be  less  than  the  cost 
of  the  analysis.  Sometimes,  of  course,  the  decision  that 
would  have  been  taken  without  this  analysis  (which  we  call 
the  "non-DA  decision")  is  not  known.  A decision  analyst 
would  maintain,  that  when  the  non-DA  decision  is  known,  its 
chances  for  being  the  same  as  that  recommended  by  the  DA 
depend,  in  part,  on  how  good  the  decision  maker  is  at  intui- 
tively integrating  into  the  decision  process  all  the  factors 
which  are  handled  explicitly  by  DA. 

The  other  statistical  difficulty  arises  from  the  qualifying 
phrase  "on  the  average  and  in  the  long  run."  A particular 
decision  may  be  correctly  identified  as  the  optimal  one  by  the 
DA  and  yet  after  the  fact  turn  out  to  be  inferior  to  some  other 
choice  of  option.  One  way  in  which  this  can  come  about  is  the 
occurrence  of  some  unfavorable,  unlikely  outcome.  This  phe- 
nomenon is  called  in  DA  the  "good  decision,  bad  outcome"  effect 
and  is  mere  generally  recognized  in  the  verse  from  Ecclesiastes: 

. . . the  race  is  not  to  the  swift,  nor  the  battle 
to  the  strong,  neither  yet  bread  to  the  wise,  nor 
yet  riches  to  men  of  understanding,  nor  yet  favor  to 
men  of  skill,  but  time  and  chance  happeneth  to  them 
all. 

The  example  of  a simple  lottery  can  be  used  to  illustrate 
this  effect.  Suppose  you  are  offered  an  opportunity  lo 
participate  once  in  a lottery  where  a ball  will  be  drawn 
from  an  urn  containing  30  blue  balls  and  70  red  balls.  If  a 
blue  ball  is  drawn,  you  lose  $20,  and  if  a red  ball  is 
drawn,  you  win  $100.  This  means  that  there  is  a 0.30  prob- 
ability of  losing  $20  and  a 0.70  probability  of  winning 
$100.  This  lottery  has  a positive  expected  value. 7 Many 


7 (0.30  x (-20))  + (0.70  x $100)  = $64. 
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people,  after  taking  into  account  their  personal  attitude 
toward  risk,  would  find  that  the  lottery  also  has  a positive 
expected  utility  for  them.  The  optimal  decision  for  anyone 
for  whom  the  lottery  has  a positive  expected  utility  is  to 
accept  the  gamble.  If  such  a decision  maker  participates  in 
the  lottery,  draws  a blue  ball,  and  loses  $20,  we  say  that  a 
bad  (unfavorable)  outcome  has  occurred.  The  initial  decision 
to  participate  would  still  be  considered  by  a decision 
analyst  to  be  the  correct  choice.®  At  the  time  the  decision 
was  made,  the  option  to  gamble  seemed  sensible;  it  is  no 
reflection  on  the  method  used  to  arrive  at  a sensible 
decision  that  the  possibility  of  losing  later  became  a 
certainty  in  this  particular  instance. 

The  implication  of  this  phenomenon  for  evaluating  DA  is 
that  one  can  neither  prove  nor  disprove  the  usefulness  of  DA 
by  looking  at  the  final  outcome  of  one  or  two  applications 
of  the  method.  A large  sample  of  similar  applications  must 
be  considered  to  gather  appropriate  statistics.  When  an 
apparent  "bad  outcome"  occurs,  however,  the  analyst  can  try 
to  get  some  idea  of  whether  its  occurrence  results  from 
vagaries  of  fortune  or  to  errors  in  the  analysis,  such  as 
incorrect  assessments  of  the  outcome  probabilities  or  of  the 
values  of  the  consequences. 

1 . 4 Indirect  Benefits  of  Decision  Analysis 

The  direct  benefit  of  DA  is  intended  to  be,  and  theo- 
retically is,  an  improvement  in  the  decision  selected.  As 
described  above,  however,  such  direct  benefit  may  not  always 
be  observed  either  because  there  is  no  change  in  the  decision 
or  because  a bad  outcome  occurs.  Whether  or  not  this  direct 
benefit  results,  however,  there  are  a number  of  indirect 
ways  in  which  the  DA  may  be  valuable.®  Two  advantages  of 


Q 

This  problem  is  discussed  by  R.  S.  Nikerson  and  C.  E.  Feehrer, 
Decision  Making  and  Training:  A Review  of  Theoretical  and 

Empirical  Studies  of  Decision  Making  and  Their  Implications 
for  the  Training  of  Decision  Makers , Technical  Report 
NAVTRAEQUIPCEN  73-C-0128-1  (August  1975),  pp.  160-164,  who 
use  the  terms  "logical  soundness"  to  indicate  a gooa  decision 
and  "effectiveness"  to  describe  a decision  having  a good 
outcome . 

9 

In  a study  of  U.S.  governmental  policy  regarding  the  export 
of  strategic  materials  to  the  Soviet  Union,  the  principal 
benefits  of  the  DA  turned  out  to  be  indirect  considerations 
such  as  the  improvement  in  communication  among  decision 
makers.  See  S.  R.  Watson  and  R.  V.  Brown,  Case  Studies  in 
the  Value  of  Decision  Analysis,  Technical  Report  75-10 
(McLean,  Virginia:  Decisions  and  Designs,  Incorporated, 

(October  1975),  pp.  3-10  through  3-14  and  A-l  through  A-4. 
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the  DA  process  are  especially  worth  considering. 

One  indirect  benefit  of  DA  is  the  improvement  in  com- 
munication when,  as  is  often  the  case  for  problems  solved 
using  DA,  more  than  one  person  is  a party  to  the  decision. 

The  process  of  DA  makes  explicit  the  important  factors  in 
the  problem,  namely,  the  options,  the  uncertainties,  and  the 
dimensions  of  value.  This  explicitness  may  reduce  or  help 
to  resolve  disagreements  among  the  people  concerned  with  the 
decision.  Without  the  format  of  DA,  two  people  might  argue 
at  length  about  which  of  two  different  decision  options 
should  be  selected  without  realizing  that  their  disagreement 
depended  upon,  for  example,  differences  of  opinion  about  the 
probabilities  of  the  various  outcomes  of  a relevant  uncertain 
event.  The  DA  format  calls  attention  to  these  figures  and 
if  necessary,  redirects  the  argument  into  the  proper  channels. 
In  order  to  restrict  disagreement  to  its  essentials,  the 
calculations  of  the  DA  can  be  repeated  by  using  different 
estimates  of  probabilities.  This  step  will  show  the  true 
effect  of  different  probability  estimates  upon  the  expected 
utility  and  the  ultimate  recommendation.  The  ability  of  DA 
to  facilitate  communication  about  a problem  is  useful  not 
only  for  resolving  differences  within  a group,  but  for 
communicating  the  justification  of  a decision  to  some  over- 
sight body  as  well. 

Another  indirect  benefit  of  DA  is  that  an  individual 
involved  in  the  analysis  learns  some  of  the  skills  needed  to 
make  correct  numerical  expressions  of  uncertainty  and  value. 
These  numbers  are  an  important  input  to  the  DA,  and  experience 
in  their  assessment  obtained  during  one  analysis  should 
improve  the  precision  of  future  analyses  involving  the  same 
decision  maker. 

Unfortunately,  these  indirect  benefits,  such  as  improved 
communication  and  experience  in  the  numerical  assessment  of 
probabilities  and  utilities,  are  rather  intangible  and,  con- 
sequently, difficult  to  quantify. 

1 . 5 Costs  of  Decision  Analysis 

The  effectiveness  of  DA  is  a measure  of  its  ability  to 
improve  decision  making.  The  efficiency  of  DA  is  a measure 
of  the  economy  with  which  this  improvement  is  obtained.  The 
efficiency  is  determined  by  comparing  the  costs  of  the 
analysis  either  with  the  benefits  of  the  improvement  or  with 
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the  costs  of  obtaining  the  improvement  by  some  other  method. 
The  major  costs  are  the  cost  of  the  analyst's  time  (the  con- 
tract price  if  the  DA  is  carried  out  by  a consulting  firm. 


as  is  often  the  case,  at  least  for  the  first  few  analyses 
done  by  an  organization)  and  the  value  of  the  time  of  people 
who  work  with  the  analyst  to  provide  the  substantive  informa- 
tion important  to  the  particular  problem  (mainly  utility  and 
probability  assessments) . Besides  these  direct  costs  of  the 
analysis,  there  may  be  some  indirect  ones  attending  the  use 
of  DA  if  it  requires  the  disruptions  and  discomfort  of 
major  departures  from  the  normal  decision-making  procedures 
of  the  organization  or  individual. 

Another  possible  large  source  of  expense  in  the  DA  is 
the  cost  of  making  a bad  decision.  If  the  whole  analysis  is 
poorly  done  or  if  some  important  factor  is  inadvertently 
omitted,  the  computed  expected  utilities  for  different 
options  may  differ  from  their  true  expected  utilities.  This 
can  lead  to  the  selection  by  the  decision  maker  of  an  option 
which  is  worse  (in  terms  of  true  expected  utility)  than  that 
which  would  have  been  chosen  without  the  analysis.  In  such 
a case,  the  cost  of  being  mistaken,  which  may  be  very  sub- 
stantial, should  be  included  when  the  analysis  is  evaluated. 

The  principal  costs  incurred  can  be  recorded  during  the 
analysis  or  recalled  at  the  end  of  the  analysis.  Indirect 
costs,  like  indirect  benefits,  can  be  estimated  at  the  end 
of  the  analysis,  but  they  are  difficult  to  quantify.  The 
cost  of  having  made  a bad  decision  generally  cannot  be  known 
until  the  important  uncertainties  are  resolved,  often  some 
time  after  the  analysis  is  completed. 

1 . 6 Biases  in  Reporting  the  Data 

Biases  in  the  reporting  of  data  needed  for  an  evaluation 
also  contribute  to  its  difficulty.  Two  such  biases  are  the 


The  possible  difference  in  efficiency  between  DA  and  other 
decision  procedures  is  shown  in  the  example  of  a military 
decision  which  was  analyzed  both  by  DA  and  by  means  of  a 
complex  simulation  model  (carried  out  by  a governmental 
laboratory) . The  recommended  decision  was  the  same  for  both 
approaches,  but  the  DA  cost  only  15%  as  much  as  the  simula- 
tion. See  J.  0.  Chinnis;  C.  W.  Kelly,  III;  R.  D.  Minckler; 
and  M.  F.  O'Connor,  Single  Channel  Ground  and  Airborne  Radio 
System  (SINCGARS)  Evaluation  Model,  Technical  Report  DT/TR 
iS-7  (McLean,  Virginia:  Decisions  and  Designs,  Incorporated, 

September  1975) . 
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distortions  due  to  hindsight  and  those  due  to  a lack  of 
candor.  Studies  of  hindsight-*-^-  have  shown,  for  example,  that 
after  the  fact  people  often  report  the  particular  outcome 
that  occurred  as  having  been  inevitable,  even  though  there 
was  considerable  prior  uncertainty  about  the  outcome.  When 
people  do  acknowledge  their  prior  uncertainty,  the  probabili- 
ties that  they  recall  to  express  uncertainty  are  often 
biased  in  a direction  that  implies  greater  foresight  than 
was  actually  the  case.  That  is,  the  outcome  that  did  occur 
is  given  high  probabilities  and  those  that  did  not,  small 
probabilities.  These  biases  can  occur  even  when  the  subject 
is  trying  to  provide  accurate  information.  Another  type  of 
bias  occurs  when  the  evaluator  consciously  withholds  or 
changes  the  data.  An  analyst  working  as  a consultant,  for 
example,  may  be  reluctant  to  admit  to  having  erred  in  an 
application  of  DA. 

1. 7 Evaluating  Other  Techniques 

Many  of  these  practical  and  statistical  difficulties  of 
evaluation,  of  course,  are  not  unique  to  the  evaluation  of 
DA.  They  are  also  encountered  in  attempts  to  evaluate  other 
decision-making  procedures  (such  as  simulation  techniques) 
and  formal  models  of  management  (in  general  systems  and 
planning,  programing,  and  budgeting  procedures.)  Managers 
and  analysts,  however,  continue  to  use  these  techniques 
despite  the  lack  of  mathematically  rigorous  proof  of  the 
value  of  particular  applications.  Two  possible  reasons  for 
the  use  of  such  techniques  are  the  appeal  of  any  apparently 
logical  format  of  the  techniques  and  the  users'  informal 
judgment  that  overall  the  benefits  of  the  method  outweigh 
its  costs. 


See  B.  Fischhoff,  "Hindsight  is  not  Equal  to  Foresight," 
Oregon  Research  Institute  Research  Bulletin  14.13  (December 
1974) , for  a discussion  of  the  distortions  of  hindsight. 
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2.0  PROPOSED  METHOD 


2 . 1  Reasons  for  Attempting  Evaluation 

The  method  proposed  here  is  designed  for  use  in  the 
retrospective  evaluation  of  DA,  despite  the  statistical 
difficulties  described  above  and  the  biases  arising  from  the 
subjective  nature  of  the  information  recollected  and  reported. 
There  are  two  main  reasons  for  carrying  out  this  evaluation. 
The  first  is  to  collect  and  summarize  available  data  on  the 
success  or  failure  of  DA  for  the  information  of  potential 
users.  These  data  can  be  summarized  in  a way  to  help  support 
or  refute  some  of  the  general  claims  and  opinions  about  DA 
which  form  the  "folklore"  of  decision  theory.  The  other 
purpose  of  the  evaluation  is  to  provide  a standard  format 
for  analysts  who  wish  to  review  their  work  in  a systematic 
way  at  the  end  of  each  specific  analysis  or  study.  This 
format  is  intended  to  help  the  analysts  to  determine  the 
strengths  and  weaknesses  of  DA  as  they  themselves  practice 
it. 


2 . 2  Possible  Approaches 

One  way  to  evaluate  an  analysis  is  simply  to  write  a 
paragraph  or  two  discussing  in  a general  way  the  degree  to 
which  the  analysis  seems  to  be  worthwhile  after  the  analysis 
is  completed.  Such  qualitative  summaries  often  appear  at 
the  end  of  articles  describing  an  application  of  DA.  Dis- 
advantages of  this  method  are  the  lack  of  uniformity  in  the 
information  reported  and  a tendency  to  stress  the  favorable 
aspects  of  the  analysis. 

At  the  opposite  end  of  the  spectrum  are  extremely 
formal  and  quantitative  approaches  to  evaluating  DA  which 
make  use  of  the  techniques  of  DA  themselves.  Matheson1 
and  Rice2  have  presented  mathematical  methods  for  calculating 
the  value  of  an  analysis  in  a given  problem  whose  structure 
has  been  determined,  expressed  in  terms  of  the  improvement 
in  the  decision  maker's  state  of  knowledge.  Another  method. 


^ J.  E.  Matheson,  "The  Economic  Value  of  Analysis  and  Computa- 
tion," IEEE  Journal  on  Systems,  Science,  and  Cybernetics 
SSC-4  (1968) : 325-32. 

2 

T.  R.  Rice,  "The  Economics  of  Decision  Making,"  Stanford 
University  Research  Report  EES-SA-74-1  (1974):  1-121. 
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proposed  by  Watson  and  Brown^,  applies  the  techniques  of  DA 
to  evaluate  a prospective  decision  analysis.  In  their 
method,  a small  decision  analysis  is  carried  out  at  several 
stages  in  the  study  of  a problem,  to  compute  the  expected 
value  of  undertaking  or  continuing  the  main  decision  analysis. 
There  are  some  possible  drawbacks  to  these  methods.  For 
example,  complex  philosophical  questions  are  raised  when 
methods  are  used  to  evaluate  themselves;  the  methods  also 
require  that  some  initial  work  be  done  before  the  study  of 
interest  to  the  user  is  attempted;  and  they  stress  direct 
benefits  and  costs  of  the  analysis  rather  than  including 
both  direct  and  indirect  factors. 

Watson  and  Brown  are  testing  the  use  of  their  method 
for  computing  the  expected  value  of  DA  on  actual  problems. 

The  method  proposed  in  this  paper  for  the  retrospective 
evaluation  of  DA  lies  between  the  two  extremes  of  a purely 
qualitative  and  a highly  quantitative  approach. 

2 . 3 Characteristics  of  the  Proposed  Method 

The  procedure  proposed  in  the  present  paper  relies  on 
anecdotal  information  about  completed  analyses;  however, 
this  information  is  to  be  collected  in  a thorough,  systematic, 
and,  to  some  extent,  quantitative  way.  It  is  proposed  that 
two  general  categories  of  information  be  collected:  informa- 

tion permittina  a macro  evaluation  of  the  steps  of  the 
procedure  (strengths  and  weaknesses  of  different  phases  of 
the  analysis) , and  information  giving  a general  description 
of  the  problem  and  its  setting. 

The  instruments  for  collecting  the  information  are 
questionnaires  to  be  filled  in  by  the  analyst  and  the  user. 

Sample  questionnaires  (Exhibits  2-1  through  2-10)  are  supplied 
for  collecting  the  data  about  a particular  DA.  Three  summary 
sheets  are  also  provided  (Exhibits  2-11  through  2-13)  on 
which  important  data  from  a number  of  studies  can  be  consoli- 
dated. The  questionnaires  establish  categories  call  for 
monetary  expressions  of  benefits  and  costs,  and  ask  open- 
ended  questions  of  the  "if  not,  why  not"  and  "give  reasons" 
type.  This  format  is  in  contrast  to  "Likert"4  type  questionnaires 


3 

S.  R.  Watson  and  R.  V.  Brown,  Issues  in  the  Value  of  Decision 
Analysis,  Technical  Report  75-9  (McLean,  Virginia:  Decisions 

and  Designs,  Incorporated,  October  1975). 

4 

R.  Likert,  "A  Technique  for  the  Measurement  of  Attitudes," 
Archives  of  Psychology  No.  140  (1932):  44-53. 
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where  the  respondent  is  given  a number  of  statements  and  for 
each  is  asked  to  express  a degree  of  agreement  by  checking 
one  of  the  following  five  responses:  strongly  disagree, 

disagree,  uncertain,  agree,  and  strongly  agree.  Nor  does 
the  proposed  format  purport  to  give  detailed,  exhaustive 
lists  of  possible  response  categories.  For  example,  the 
degr  :e-cf-use  questionnaire  in  Exhibit  2-2  provides  five 
possible  reasons  for  management's  failure  to  follow  the 
recommendations  of  a DA.  In  a large-scale  study  of  the 
implementation  of  operations  research  and  management  science 
models,  Sorenson  and  Zand^  designed  a questionnaire  in  which 
sixty  reasons  are  proposed  for  the  failure  to  implement  the 
results  of  a study.  These  latter  types  of  questionnaires 
are  not  appropriate  for  the  evaluation  methodology  presented 
here  because  of  the  small  proposed  sample-size  (about  20 
studies) , the  large  number  of  variables  of  interest,  and  the 
exploratory  nature  of  this  evaluation  methodology. 

2 . 4 Macro  Evaluation 

A number  of  questions  can  ,be  investigated  or  answered 
in  the  macro  evaluation  of  a number  of  DA  applications. 

These  research  questions  are  not  stated  in  terms  of  hy- 
potheses because  of  the  present  impossibility  of  collecting 
enough  information  to  make  rigorous  statistical  tests  of 
such  hypotheses. 

2.4.1  Level  of  completion  - In  what  percentage  of  the 
cases  studied  was  the  DA  completed?  For  those  not  completed, 
at  what  stage  and  for  what  reason  were  they  abandoned? 

An  analysis  may  be  abandoned  well  before  the 
expected  value  or  utility  of  different  decision  options  is 
computed.  The  reason  may  fall  into  one  of  three  main  cate- 
gories. The  first  of  these  may  tend  to  confirm  the  usefulness 
of  the  DA  method,  namely,  the  optimal  decision  option  becomes 
obvious  during  the  structuring  of  the  problem  (as  the  options 
are  listed  and  the  uncertainties  and  dimensions  of  value 
identified)  or  during  the  early  stages  of  the  probability 
assessment.  Some  people  believe  that,  for  many  problems, 
the  majority  of  the  benefits  of  DA  accrue  during  the  early. 


R.  L.  Sorensen  and  D.  E.  Zand,  "Improving  the  Implementation 
of  OR/MS  Models  by  Applying  the  Lewin-Schein  Theory  of 
Change , " Implementing  Operations  Research/Management  Science, 
eds.  R.  L.  Schultz  and  D.  p"l  Slevin  (New  York:  American 

Elsevier  Publishing  Co.,  1975),  p.  230. 
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structuring  stages  of  the  work.  This  view  holds  that  the 
principal  advantage  of  DA  is  its  requirement  that  the 
important  factors  of  the  decision  be  expressed  in  an  orderly 
manner,  rather  than  its  use  of  maximum  expected  utility  as  a 
decision  factor.  If  many  analyses  fall  into  this  category 
of  abandonment,  then  some  support  will  be  given  to  that 
claim.  The  descriptive  information  (discussed  in  Section 
2.6)  will  provide  some  indication  of  the  type  of  problem  or 
organization  for  which  this  is  the  case  (e.g.,  first-time 
users  or  few  dimensions  of  value) . 

The  second  category  contains  problems  whose 
analysis  is  discontinued  because  the  analysis  is  badly  done 
(possibly  because  the  problem  was  never  suited  for  this  kind 
of  study) . A third  catch-all  category  of  causes  for  curtailing 
a DA  is  "reasons  unrelated  to  the  DA"  (e.g.,  the  sponsoring 
group  loses  responsibility  for  the  decision  being  studied.) 

Exhibit  2-1  shows  a questionnaire  that  can  be 
used  to  collect  this  information.  The  responses  to  "if  not, 
why  not"  are  then  sorted  into  the  categories  described 
above . 

2.4.2  Degree  of  use  - In  what  percentage  of  the  cases 
studied  was  the  recommended  decision  actually  taken? 

The  crucial  test  of  the  value  of  a DA  must  be 
whether  or  not  the  recommendation  of  tfee^analysis  is  followed. 
An  interesting  statistic,  therefore,  is  the  percentage  of 
analyses  in  which  the  recommended  decision  option  is  adopted. 
This  figure,  like  many  others  in  the  study,  may  be  biased 
because  successful  DA's  are  more  likely  to  be  reported  than 
unsuccessful  ones.  When  the  recommended  decision  is  not 
taken,  an  attempt  is  made  to  identify  reasons  for  the  rejection 
of  the  results  of  the  analysis.  Some  sample  reasons  are 
provided  in  Exhibit  2-2. 

There  is  a tendency  among  the  people  who  carry 
out  DA  to  lose  track  of  the  problem  studied  as  soon  as  the 
results  of  the  analysis  are  delivered.  This  inquiry  into 
the  use  of  the  results  will  help  determine  the  truth  of 
rumors  that  many  careful  and  elaborate  analyses  are  never 
implemented. 

2.4.3  Comparisons  among  costs  and  benefits  - How  do 
the  costs  and  expected  benefits  of  th,e  decision  option 
recommended  by  the  DA  compare  to  the  costs  and  expected 
benefits  of  the  option  selected  by  another  method?  How  do 
expected  costs  compare  with  actual  costs;  estimated  benefits 
with  actual  benefits? 


1 

LEVEL- OF- COMPLETION  QUESTIONNAIRE 

1.  Was  the  DA  completed? 

Yes  No  

If  not,  why  not? 

Optimal  decision  option  became  obvious  early 

in  the  DA. 

Method  turned  out  to  be  inappropriate  for  the 

problem.  Give  reasons:  


User  became  impatient  with  the  amount  of  time 
and  effort  required  by  the  DA. 

Resources  (time,  money,  etc.)  exhausted  before 
completion. 

Other.  Specify:  



EXHIBIT  2-1 
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DEGREE-OF-USE  QUESTIONNAIRE 


What  was  the  option  recommended  by  the  analysis? 


Was  this  decision  taken? 

Yes  No  

If  not,  why  not? 

Results  of  DA  provided  too  late. 

DA  incorrect  (important  factors  omitted  or 
other  errors).  Specify: 


User  did  not  understand  the  DA. 

"Political"  factors  intervened  to  prevent  the 
choice  of  the  optimal  decision. 

Other.  Specify:  


EXHIBIT  2-2 


The  most  relevant  comparison  of  the  costs  and 
benefits  of  the  DA  is  with  those  costs  and  benefits  which 
would  have  been  incurred  without  the  analysis  (or  with  some 
other  type  of  analysis) . Unfortunately,  these  figures  may 
be  unavailable  or  very  difficult  to  estimate. ^ The  first  six 
columns  of  Exhibit  2-3  indicate  the  information  needed  to 
carry  out  a full  computation  of  the  net  expected  value  of 
the  analysis.  The  net  expected  value  is  defined  as  the 
total  difference  in  expected  benefit  (between  the  DA  and  the 
other  method)  less  the  total  difference  in  cost  of  the  DA 
and  the  other  method.  The  crucial  information  for  purposes 
of  comparison  here  is  data  about  what  the  cost  and  expected 
benefits  would  have  been  if  the  DA  had  not  been  undertaken. 

In  attempting  to  collect  these  data,  the  evaluator  encounters 
one  of  several  situations.  These  are: 

(1)  Status  quo  case:  If  without  the  DA  no  action  would 

have  been  taken,  the  expected  costs  and  benefits 

of  doing  nothing  can  be  taken  as  the  base  from 
which  anything  else  will  be  measured  and  so  set  to 
zero.  Thus,  the  "other  method"  columns  under  cost 
and  benefit  will  contain  only  zero  values.  Then 
the  expected  benefit  under  the  option  recommended 
by  the  DA  can  be  isolated  and  attributed  entirely 
to  the  DA.  The  direct  and  indirect  costs  of  the 
DA  are  accumulated  during  the  analysis.  The  net 
expected  of  the  analysis  is  simply  the  cost  of  the 
DA  minus  zero  subtracted  from  the  expected  value 
of  the  decision  option  recommended  by  the  DA  minus 
zero. 

(2)  Unaided  decision  known:  The  decision  option  that 

would  have  been  taken  without  the  DA  is  known  for 
certain.  The  expected  benefits  computed  by  the  DA 
for  the  two  decision  options  can  then  be  compared 
(assuming,  as  is  generally  the  case,  that  the 
option  selected  by  the  unaided  decision  is  also 


6This  problem  of  making  reasonable  deductions  conditional 
on  a false  premise  ("If  the  DA  had  not  been  done,  what 
would  have  happened?")  is  discussed  by  logicians  as  the 
topic  of  "counterf actuals . " See,  for  example,  D.  Lewis, 

Coun ter f actuals  (Cambridge,  Massachusetts:  Harvard  University 

Press , 197?)  , pp.  1-142. 


included  as  an  option  in  the  DA. ) ^ The  costs  of 
the  DA  are  found  as  in  situation  (1) . If  the 
unaided  decision  option  is  known,  then  presumably 
the  cost  for  arriving  at  that  decision  is  known  or 
can  be  easily  estimated.  All  the  information  for 
columns  one  through  six  of  Exhibit  2-3  is  therefore 
available.  (In  the  special  case  where  the  option 
selected  by  the  unaided  decision  and  that  recommended 
by  the  DA  are  the  same,  the  data  in  the  benefits 
columns  will  be  the  same  for  "DA  method"  and 
"other  method.")  The  differences  in  cost  between 
the  two  methods  can  be  calculated,  as  can  the 
differences  in  expected  benefit.  The  net  expected 
value  of  the  DA  is  the  total  difference  in  expected 
benefit  less  the  total  difference  in  costs. 

(3)  Several  possible  unaided  decision  options:  In 

situations  where  the  unaided  decision  option  is 
not  known  for  certain,  it  may  be  that  the  evaluator 
can  obtain  probability  assessments  of  the  likelihood 
that  each  of  the  various  different  candidate 
options  would  have  been  chosen  without  the  analysis. 
The  "other  method"  columns  in  Exhibit  2-3  (columns 
2 and  5)  can  then  be  filled  with  costs  and  expected 
benefits  which  are  weighted  averages  of  the  costs 
and  expected  benefits  for  each  of  the  candidate 
decision  options.  (The  weights  are  the  probabilities 
that  each  option  would  have  been  chosen.)  Usually 
the  decision  option  selected  by  the  DA  will  be 
included  as  one  of  the  candidate  options  for  the 
unaided  method;  however,  in  some  instances  this 
decision  option  may  be  one  generated  by  the  DA  and 
which  would  not  have  been  considered  otherwise. 

The  expected  benefit  for  each  candidate  option  can 
be  obtained  from  the  DA  since  these  candidates 
would  normally  all  be  included  in  the  DA.  On  the 
other  hand,  the  costs  of  arriving  at  each  of  these 
decisions  by  some  method  other  than  DA  may  be 
difficult  to  estimate. 


In  a study  of  possible  programs  for  the  commercialization 
of  synthetic  fuels  in  which  the  Decision  Analysis  group  at 
Stanford  Research  Institute  carried  out  a DA,  the  difference 
in  expected  benefit  between  the  expected  unaided  decision  and 
that  recommended  by  the  DA  was  approximately  $3.8  billion. 

See  Synthetic  Fuels  Commercialization  Program,  Vol.  II,  Cost/ 
Benefit  Analysis  of  Alternative  Production  Levels,  Report 
Submitted  by  Synfuels  Interagency  Task  Force  to  the  President's 
Energy  Resources  Council  (November  1975) . 
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(4)  Unaided  decision  unknown:  In  some  cases  the 

analyst  or  evaluator  may  have  no  idea  about  what 
option  would  have  been  taken  without  the  DA  or 
even  what  alternatives  would  have  been  considered. 

In  this  case  the  net  expected  value  of  the  DA 
would  be  estimated  by  considering  only  the  costs 
of  the  DA  in  comparison  to  the  expected  benefits 
of  its  recommended  decision  option  (column  4 minus 
column  1 in  Exhibit  2-3.)  This  limited  assessment 
of  the  expected  value  of  the  DA  probably  overestimates 
the  value  since  the  calculation  reduces  to  that  of 
situation  (1)  where  it  is  assumed  that  the  alter- 
native method  to  the  DA  has  no  cost  and  yields  no 
benefit.  (If  the  unknown  expected  benefit  of  the 
other  method  is  greater  than  its  unknown  cost, 
the  net  expected  value  of  the  DA  will  be  too  high.) 

The  net  expected  value  of  the  DA  is  a measure  that 
compares  how  well  the  analyst  expects  to  do  by  carrying  out 
a DA  versus  using  some  other  method.  As  discussed  earlier, 
however,  the  expected  gains  of  an  analysis  may  not  be  realized 
if,  for  example,  the  uncertainty  is  resolved  in  an  unfavorable 
manner  (if,  in  other  words,  a "bad  outcome"  occurs.)  A 
second  interesting  comparison,  therefore,  is  between  the 
expected  benefit  under  the  decision  option  recommended  by 
the  DA  and  the  actual  benefit  if  that  decision  is  taken. 
Differences  between  actual  and  expected  benefits  can  also 
occur  if  some  part  of  the  analysis  is  incorrectly  done. 

As  part  of  the  evaluation,  data  are  collected  to 
show  how  often  the  DA  changes  the  decision  to  be  taken,  how 
often  the  actual  and  expected  benefits  of  the  recommended 
decision  differ,  what  the  reasons  are  for  the  difference, 
and  how  much  they  differ.  The  last  three  columns  of  the 
questionnaire  in  Exhibit  2-4  provide  a means  for  recording 
the  actual  benefits.  Since  only  one  option  is  taken,  the 
figures  will  be  the  actual  observed  benefits  in  one  column 
but  for  the  other  column  will  be  estimates  of  what  would 
have  been  observed  had  the  other  decision  option  been  taken. 

Exhibit  2-5  shows  a questionnaire  designed  for 
collecting  information  about  how  often  and  why  the  expected 
and  actual  benefits  differ.  The  data  about  indirect  costs 
and  benefits  must  be  collected  and  reduced  to  monetary  terms 
for  use  in  determining  the  net  value  shown  in  Exhibit  2-3. 

This  part  of  the  evaluation  may  be  quite  difficult  for  some 
users  or  in  some  applications.  The  questionnaires  shown  in 
Exhibits  2-6  and  2-7  give  a possible  breakdown  of  the  desired 
information  but  do  not  solve  the  problem  of  how  the  transformation 
to  monetary  terms  is  to  be  accomplished. 
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AIDED  VERSUS  UNAIDED  OPTION  QUESTIONNAIRE 

1.  What  is  the  option  recommended  by  the  DA? 


2.  What  option  would  have  been  taken  without  the  DA?  Was  it 
Known?  (Specify)  


Uncertain?  (Give  possibilities  and  their  proba- 
bilities of  having  been  selected) 

Option  Probability 


Unknown? 


3.  If  known,  is  it  the  same  or  different  from  that  recommended 


by  the  analysis? 


EXHIBIT  2-4 
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ACTUAL  AND  EXPECTED  BENEFITS  QUESTIONNAIRE 


Was  the  recommendation  of  the  DA  followed? 

Yes  No  (if  no,  the  "actual"  benefits 

are  estimates) 

Did  actual  benefits  of  the  DA  differ  from  the  expected 
benefits? 

Yes  No 

If  yes,  how  much  was  the  difference  (as  a percentage  of 
the  total  benefit) ? 

Was  the  difference 

Favorable  (greater  benefits  realized)  or 

Unfavorable  (few  actual  benefits) 

What  are  the  reasons  for  the  difference? 

"Bad  outcome"  (give  the  probabilities  assessed  for 

the  desired  outcome)  

the  outcome  which  occurred  ) 

Incorrect  analysis  (list  the  mistakes  in  the 

analysis) 


EXHIBIT  2-5 


INDIRECT  COSTS  OE  THE  DECISION  ANALYSIS 


Check  the  types  of  costs  incurred  in  the  analysis.  Estimate 
the  amount  of  these  costs  in  any  natural  units  and  then 
estimate  the  corresponding  monetary  value. 


Monetary 

Cost  Type  Amount  Equivalent 

1.  Time  of  employees 
working  with  the 

analyst  days  = $ 


2.  Others  (specify) 


3. 


4. 


EXHIBIT  2-6 
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INDIRECT  BENEFITS  OF  THE  DECISION  ANALYSIS 


Check  the  types  of  benefits  obtained  and  estimate  their 
value  in  monetary  terms. 


Benefit  Type  Monetary  Equivalent 


1.  Improved  communication 
among  group  decision 
makers . 


2.  Ease  of  obtaining  approval 
of  the  decision  by  higher 
levels  in  the  organization. 


3.  Same  model  can  be  applied 
with  minor  changes  to 
other  problems. 


4.  Practice  in  assessing 

probabilities  and  utilities. 


5.  Other  (specify) 


EXHIBIT  2-7 


2 . 5  Micro  Evaluation 


During  the  micro  evaluatior  of  a DA,  each  of  the  main 
steps  in  the  DA  is  examined  in  turn  to  see  how  well  that 
part  of  the  analysis  was  performed.  The  steps  considered  are: 
structuring  the  problem,  assessing  the  probabilities  and 
utilities,  computing  the  expected  utility  of  each  option  to 
find  the  one  with  the  highest  value,  and  testing  the  sensitivity 
of  the  recommendation  of  the  DA  to  changes  in  structure  and 
inputs.  First,  the  analyst  is  asked  to  recall  which  steps 
seemed  most  helpful  during  the  analysis  by  revealing  important 
information  which  had  not  been  considered  before  the  DA  was 
undertaken.  Next,  the  analyst  evaluates  these  steps  in  light 
of  all  that  is  known  by  the  end  of  the  analysis  and  after  some 
of  the  uncertainty  has  been  resolved.  The  questions  to  be 
answered  at  this  point  are:  what  important  factors  were  com- 

pletely overlooked  and  what  inputs  (probabilities  or  utilities) 
were  overestimated  or  underestimated.  Exhibit  2-8  presents  a 
format  in  which  these  data  can  be  collected. 

For  each  example  of  a DA,  these  data  provide  information 
about  which  steps  in  the  analysis  were  most  helpful.  This 
determination  may  depend  strongly  on  the  particular  kind  of 
problem  analyzed  (the  relevant  categories  of  problem  have  yet 
to  be  specified)  or  on  the  amount  of  experience  of  the  analyst. 
The  user  of  the  recommendation  of  the  DA  may  also  be  asked  which 
aspects  of  the  analysis  seemed  to  be  the  most  difficult,  the 
easiest,  the  most  helpful,  and  so  on. 

2 . 6 Problem  Description 

The  purpose  of  the  problem  description  section  of  the 
evaluation  is  to  collect  information  about  variables  in  the 
problem  that  may  turn  out  to  be  correlated  with  the  overall 
success  of  the  DA  or  with  the  detailed  evaluation  of  the  dif- 
ferent steps  in  the  DA.  These  descriptive  data  deal  with  the 
magnitude  of  the  problem,  the  amount  of  resources  at  risk, 
the  analyst's  and  the  user's  level  of  experience  with  DA,  and 
the  type  of  organization  making  the  decision.  Exhibit  2-9 
provides  a format  for  collecting  this  information. 

2 . 7 Summary  Sheets 

After  a number  of  DA's  have  been  evaluated  by  using  the 
questionnaires  shown  in  Exhibits  2-1  through  2-9,  the  results 
of  this  sample  survey  can  be  summarized  in  the  format  presented 
in  Exhibits  2-10  through  2-13. 
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MICRO  EVALUATION  QUESTIONNAIRE 


AFTER  THE  ANALYSIS 


DURING  THE  ANALYSIS 


Note  the  contribution  of  each  step  in  the  analysis 
to  the  description  and  resolution  of  the  problem 
during  the  analysis.  Next  rank  these  contributions 
in  order  of  their  overall  importance  to  the  DA. 
(These  data  give  an  indication  of  how  much  better 
DA  is  than  a traditional  method  of  problem 
solving) . 


Note  for  each  step  in  the  analysis  what  important  factors 
were  omitted  and  what  inputs  were  incorrectly  assessed. 
(These  data  indicate  how  the  DA  as  performed  compares  to 
a perfect  DA) . 


Any  important  elements  omitted? 
(list  them,  with  reason  omitted 


important  elements  revealed  during 
:he  DA  (list  them) 


STRUCTURE 


Generating  options 


Identifying 

Probabilities 


Identifying  dimensions 
of  values 


Were  Assessments 


If  yes,  were  they 
Under  lOver 
Estimated lEstimated 


Were  Assessments 
Incorrect? 

No  | Yes 


Probabilities 


Weighting  factors  for 
different  dimensions 
of  value 


Conversion  to  utilities! 


SENSITIVITY 


Which  assumptions  or  inputs  were 
shown  by  the  sensitivity  analysis 
to  be  important?  (specify) 


Which  critical  assumptions  or  inputs  did  the  sensitivity 
fail  to  detect  and  why? 


Probabilities 


Incremental  costs  and 
Benefits 


Weighting  factors  for 
different  dimensions 
of  value 


Conversion  to  utilities 


EXHIBIT  2-8 


DESCRIPTION  OF  THE  DECISION  PROBLEM 


Resources  at  risk: 


1.  What  is  the  amount  ($)  at  risk  in  this  problem? 

$ (or  range) 


2.  What  percentage  is  this  of  the  annual  budqet  of  the 
organizational  unit  making  the  decision? 

% (or  range) 


Magnitude 


3.  a. 
b. 

4 . a. 
b. 

5 . a. 
b. 


and  complexity  of  the  problem: 

Number  of  decision  nodes  . 

Total  number  of  options  (summed  over  all 
decision  nodes)  . 

Number  of  uncertain  events  considered  

Total  number  of  different  event  outcomes  

Number  of  dimensions  of  value  

Total  number  of  outcomes  evaluated  (summed  over 
all  different  dimensions)  


Experience  of  analyst  and  user: 

6.  How  much  experience  has  the  analyst  had  with  this 

type  of  problem?  

7.  How  many  DA's  has  the  user  participated  in?  

8.  How  much  did  the  user  participate  in  this  DA? 


Type  of  organization  in  which  the  decision  will  be  made. 


9. 

Profit  , non-profit 

, governmental 

(give  agency 

) 

10. 

Annual  sales  of  organization 

11. 

Management  style 

centralized 

decentralized. 

EXHIBIT  2-9 
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EXHIBIT  2-11 


I 


SUMMARY  OF  THE  MICRO  EVALUATION 
STATISTICS  OVER  THE  WHOLE  SAMPLE 


DURING  ANALYSIS,  PERCENTAGE  OF 
TIME  EACH  STEP  WAS  IDENTIFIED  AS 


Revealing  Important  New  Elements 


Incremental  Costs 
and  Benefits 


ting  Factors 
if  ferent 
sions  of 


Conversion  to 
Utilities 


Tn  each  column,  circle  the  largest  percentage 


Sample  Size 


EXHIBIT  2-12 
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SUMMARY  OF  THE  DESCRIPTIVE  INFORMATION 
ABOUT  PROBLEM  SIZE- 


AMOUNT  AT  RISK 


NUMBER  OF 


PERCENTAGE  OF 

APPLICATION  DOLLARS  ANNUAL  SALES 


DECISION  EVENT  VALUE 
NODES  NODES  DIMENSIONS 


1. 

2. 

3. 

4 . 

5. 

6. 

7. 

8. 

9. 

10. 

11. 

12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 

20. 

Average : 


EXHIBIT  2-13 


4 
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3.0  DISCUSSION 


The  purposes  of  this  evaluation  procedure  are  to  estab- 
lish a record  of  the  long-run  gains  resulting  from  the  use 
of  DA  (in  general,  or  by  a particular  analyst  or  organization) 
and  to  encourage  users  and  analysts  to  review  their  work  on 
a regular  basis  at  the  end  of  each  analysis. 

It  may  be  that  the  procedure  recommended  here  is  too 
much  trouble,  requiring,  as  it  does,  that  the  analyst  and 
user  spend  time  thinking  about  a problem  after  the  DA  has 
supposedly  solved  it.  People's  tendency  at  the  end  of  a 
successful  quantitative  study  is  briefly  to  express  pleasure 
with  the  outcome  and  then  look  for  another  possible  applica- 
tion of  the  method.  If,  on  the  other  hand,  the  study  has 
turned  out  badly,  people  tend  to  put  the  unfavorable  experience 
out  of  their  minds.  For  a methodology  of  evaluation  to 
overcome  these  tendencies,  the  evaluator  must  believe  that 
the  information  demanded  can  be  produced , that  it  is  valuable, 
and  that  the  sample  questionnaires  are  an  efficient  way  of 
collecting  the  information.  These  two  points  are  presently 
being  investigated  by  cipplying  parts  of  the  methodology  to  a 
number  of  past  DA  studies. 

A more  serious  problem  is  the  overall  validity  of  the 
data.  Can  the  indirect  costs  and  benefits  be  computed  in 
any  reasonable  way?  Are  the  judgments  about  what  would  have 
happened  without  the  application  of  DA  nonsense  because  of 
problems  of  hindsight  and  lack  of  candor?  Full-scale  applica- 
tion of  the  evaluation  methodology  is  proposed  to  help 
answer  these  questions.  If  distortions  because  of  hindsight 
seem  to  be  a serious  problem,  they  can  be  lessened,  for 
example,  by  having  the  users  of  the  DA  record  formally 
before  the  analysis  begins  their  initial  impressions  about 
such  things  as  the  probabilities  and  the  decision  which 
would  be  chosen  without  benefit  of  the  DA.  Frankness  in 
assessing  the  factors  used  to  determine  the  retrospective 
value  of  an  analysis  can  perhaps  be  encouraged  by  reporting 
the  results  in  an  anonymous  fashion. 
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